Systems and methods for generating and using semantic images in deep learning for classification and data extraction

ABSTRACT

Disclosed is a new document processing solution that combines the powers of machine learning and deep learning and leverages the knowledge of a knowledge base. Textual information in an input image of a document can be converted to semantic information utilizing the knowledge base. A semantic image can then be generated utilizing the semantic information and geometries of the textual information. The semantic information can be coded by semantic type determined utilizing the knowledge base and positioned in the semantic image utilizing the geometries of the textual information. A region-based convolutional neural network (R-CNN) can be trained to extract regions from the semantic image utilizing the coded semantic information and the geometries. The regions can be mapped to the textual information for classification/data extraction. With semantic images, the number of samples and time needed to train the R-CNN for document processing can be significantly reduced.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims a benefit of priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 17/081,705,filed Oct. 27, 2020, entitled “SYSTEMS AND METHODS FOR GENERATING ANDUSING SEMANTIC IMAGES IN DEEP LEARNING FOR CLASSIFICATION AND DATAEXTRACTION,” which is a continuation of, and claims a benefit ofpriority under 35 U.S.C. § 120 from, U.S. patent application Ser. No.16/842,097, filed Apr. 7, 2020, entitled “SYSTEMS AND METHODS FORGENERATING AND USING SEMANTIC IMAGES IN DEEP LEARNING FOR CLASSIFICATIONAND DATA EXTRACTION,” issued as U.S. Pat. No. 10,832,048, which is acontinuation of, and claims a benefit of priority under 35 U.S.C. § 120from, U.S. patent application Ser. No. 16/058,476, filed Aug. 8, 2018,entitled “SYSTEMS AND METHODS FOR GENERATING AND USING SEMANTIC IMAGESIN DEEP LEARNING FOR CLASSIFICATION AND DATA EXTRACTION,” issued as U.S.Pat. No. 10,628,668, which claims a benefit of priority under 35 U.S.C.§ 119(e) from U.S. Provisional Application No. 62/543,246, filed Aug. 9,2017, entitled “SYSTEMS AND METHODS FOR GENERATING AND USING SEMANTICIMAGE.” All applications listed in this paragraph are fully incorporatedby reference herein for all purposes.

TECHNICAL FIELD

This disclosure relates generally to document processing utilizingmachine learning. More particularly, this disclosure relates to systemsand methods for generating semantic images from input images ofdocuments and utilizing the semantic images to improve the training andperformance of a machine learning engine for classification and dataextraction.

BACKGROUND OF THE RELATED ART

It is often necessary for companies to extract information from varioustypes of documents such as invoices, purchase orders, correspondences,etc., and to enter this information in their own data systems so thatthey can use the information for various enterprise operations andprocesses. While the process of extracting document information,converting the extracted document information into a computer-usableform and entering the converted information into a data processingsystem was historically performed manually, computer-based systems havebeen developed to automate this process. Computer-based systems areavailable, for example, to perform optical character recognition (OCR)on images of scanned documents and to thereby generate digital data fromthe images. The strings of recognized characters can be processedaccording to a predetermined set of algorithms to identify informationthat is represented by the character strings. OCR techniques are knownto those skilled in the art and thus are not further described herein.

While automated data extraction systems can reduce the burden of havingto manually identify and re-enter the information that is found inbusiness documents, these systems still have some drawbacks. Forinstance, traditionally, it has been necessary to program these systemswith an extensive set of rules that are used to determine the type ofinformation that is represented by a particular string of characters.For example, a rule may be needed to determine that a character string“New York” represents a city name, or a rule may be needed to determinethat a character string “Smith” represents a person's surname. Thisprogramming requires a great deal of time and effort, and any exceptionto the predefined rules may “break” the algorithm and require specialhandling. In addition to the high cost of programming these systems, itis usually necessary to perform manual validation of the resultsgenerated by the systems, which further increases the cost.

SUMMARY OF THE DISCLOSURE

Embodiments disclosed herein can address the aforementioned drawbacksand provide additional technical solutions and benefits. Theseembodiments advantageously leverage the power of deep learning (DL) toimprove the performance of machine learning (ML) engines forclassification and data extraction from document images, and to reduceor eliminate the programming that was previously required to implementtraditional algorithmic rules used in training ML engines. Theincorporation of deep learning enables a ML engine to learn andclassify/extract by examples, rather than by rules. While training a MLengine usually requires a large number of training samples and a greatdeal of training time, embodiments disclosed herein make use of semanticimages to facilitate the training and the performance of the ML enginein classification and data extraction tasks.

In some embodiments, a method for generating and using semantic imagesmay include receiving an input image of a document, generating textualinformation with geometries from the input image, converting the textualinformation with geometries to semantic information, generating asemantic image coded with the semantic information, extracting regionsfrom the semantic image through a ML engine enhanced with a DL feedbackloop, and extracting data from the regions provided by the ML engine.The method can be implemented on a server computer operating in anenterprise computing environment. The server computer can be part of anenterprise system. Such an enterprise system can be capable of capturingdocuments and data from paper, electronic files, and other sources,transforming the captured documents and data into digital content, andcommunicating the transformed digital content to downstream computingfacility or facilities, for instance, content management systems,enterprise operations and processes, and so on.

In some embodiments, text with geometries can be generated from an inputimage of a document by an OCR module. In some embodiments, the inputimage may already have text with geometries when received by a systemimplementing the method. In such cases, the method can proceed toconvert the text bounded by the geometries (e.g., bounding boxes,rectangles, frames, etc.) to semantic information.

In some embodiments, the system includes a semantics converter moduleconfigured for converting the text to semantic information (i.e., themeaning or semantic type of the text) utilizing a language knowledgebase. For example, the semantics converter module is operable todetermine that a piece of text bounded by a geometry contains a textstring “July”. The semantics converter module can access and search thelanguage knowledge base for information on “July” and determine that thesemantic type of “July” is “month”. Rather than extracting the actualcontent of the text string “July”, the semantics converter module isoperable to determine a code for the text string based on its semantictype “month”.

Any coding schemes suitable for identifying different semantic types canbe used. “Coding”, in this sense, can include finding, from the languageknowledge base, a data type or semantic type associated with the textunder evaluation and writing, for instance, a set of pixelscorresponding to the text under evaluation in a certain code or colorthat corresponds to the semantic type. For example, the semantic type“month” may be coded as “1” or “yellow” with a set of x-y coordinates(relative to the document); the semantic type “day” may be coded as “2”or “red” with another set of x-y coordinates; the semantic type “year”may be coded as “3” or “orange” with yet another set of x-y coordinates;and so on. The finding can be done utilizing a dictionary or reverseindex of the language knowledge base and the pixel writing can be doneutilizing a painting tool, facility, or module. The number of semantictypes corresponds to the number of semantic layers in an output semanticimage. For example, if there are four semantic types, there can be fourcolors or codes, indicating four semantic layers in an output semanticimage. Codes in the programming sense (as opposed to colors forvisualization by humans) can be handled more efficiently by algorithms.

In some embodiments, the semantics converter module is further operableto generate a semantic image based on the semantic types that can bedetermined from the input image. In this disclosure, a “semantic image”refers to an image of a document that is coded with the semanticinformation of the document without any textual content of the document.Such a semantic image has the location information where the semanticinformation can be found on the document and has codes corresponding tothe semantic information by type.

In some embodiments, the semantic image may be provided as input to a MLengine implementing a specifically trained artificial neural network(ANN). In some embodiments, the ML engine implements a region-basedconvolutional neural network (R-CNN) and operates as an R-CNN runtimeengine.

In some embodiments, the R-CNN runtime engine is operable to process thesemantic image and extract regions from the semantic image for textmapping. For instance, responsive to “month”, “day”, and “year” beingfound in close proximity to each other in the semantic image (based onthe associated location information such as x-y coordinates), the R-CNNruntime engine can identify them as a region for a date. As anotherexample, responsive to certain semantic types (e.g., “number”, “street”,“city, state”, and “zip code”) being found in close proximity to eachother, the R-CNN runtime engine can identify them as a region for anaddress. Here, “date” and “address” are examples of data fields that canbe processed by a text mapping module downstream from the R-CNN runtimeengine.

In some embodiments, processing outputs from the R-CNN runtime engineare provided to a DL feedback loop. The DL feedback loop includes atraining process that can update/improve a field type knowledge basewhich is utilized by a DL learner. The DL learner learns how to look forregions in semantic images and produces R-CNN parameters (e.g., minimumerror, number of neurons, etc.). The R-CNN parameters can then be fedback to the R-CNN runtime engine to improve its performance and outputs.

In some embodiments, a text mapping module is operable to utilize theregions outputted by the R-CNN runtime engine and the text withgeometries from the input image to identify, classify, and/or extractdesired information from the document. For example, responsive toreceiving an input image of an invoice, the system can quickly,intelligently, and automatically identify semantic types such as“street”, “city”, “state”, “month”, “day”, “year”, “number”, etc.contained in the invoice, generate a semantic image of the invoice codedwith the semantic types, identify and extract regions such as “address”,“invoice date”, “invoice number”, “purchase order number”, “tableheader”, “table body”, and “invoice totals” from the semantic image, andextract data from each region for a corresponding data field. In thisway, the system can understand the unstructured information contained inthe input image and intelligently transform the unstructured informationinto structured data that can be consumed by a downstream computingfacility or facilities.

Identifying semantic information by their respective semantic typesallows different types of semantic information be treated in the sameway. For example, states “Texas”, “TX”, “New Mexico”, “NM”,“California”, “CA”, “New York”, “NY”, and character stringscorresponding to all the other states can be classified by the singlesemantic type “state name” and given a single code for the semantic typein a semantic image. Likewise, different addresses can be coded to thesame semantic type “address” and different variants of “month”, “day”,“year” (e.g., “July 31, 2018”, “31 July 2018”, “07/31/2018”, “7/31/18”,etc.) can be coded to the same semantic type “date”. In this way, theR-CNN does not need to be trained to handle all the detailed textualinformation, such as all the street numbers, street names, cities,states, and zip codes and their variants, all the “year”, “month”, “day”and their variants, and so on, in input images. Instead, the R-CNN canbe trained to recognize regions, which correspond to desired fields suchas “state name”, “address”, “date”, etc., from semantic images. Sincethe R-CNN does not need to be trained to handle all the detailed textualinformation, the number of samples and the time and computational costsneeded for training the R-CNN can be significantly reduced.

One embodiment comprises a system comprising a processor and anon-transitory computer-readable storage medium that stores computerinstructions translatable by the processor to perform a methodsubstantially as described herein. Another embodiment comprises acomputer program product having a non-transitory computer-readablestorage medium that stores computer instructions translatable by aprocessor to perform a method substantially as described herein.Numerous other embodiments are also possible.

These, and other, aspects of the disclosure will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. It should be understood,however, that the following description, while indicating variousembodiments of the disclosure and numerous specific details thereof, isgiven by way of illustration and not of limitation. Many substitutions,modifications, additions, and/or rearrangements may be made within thescope of the disclosure without departing from the spirit thereof, andthe disclosure includes all such substitutions, modifications,additions, and/or rearrangements

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the invention. A clearerimpression of the invention, and of the components and operation ofsystems provided with the invention, will become more readily apparentby referring to the exemplary, and therefore non-limiting, embodimentsillustrated in the drawings, wherein identical reference numeralsdesignate the same components. Note that the features illustrated in thedrawings are not necessarily drawn to scale.

FIG. 1 depicts a diagrammatic representation of a data classificationand extraction system according to some embodiments.

FIG. 2 depicts a diagrammatic representation of a language knowledgebase according to some embodiments.

FIG. 3 depicts a diagrammatic representation of a field type knowledgebase according to some embodiments.

FIG. 4 depicts a flow diagram illustrating an R-CNN training processaccording to some embodiments.

FIG. 5 depicts a diagrammatic representation of a ML training processwithout a DL feedback loop.

FIG. 6 depicts a diagrammatic representation of a ML training processwith a DL feedback loop according to some embodiments.

FIG. 7 depicts a flowchart illustrating an exemplary method inaccordance with some embodiments.

FIGS. 8A-8E depicts a diagrammatic representation of the transformationof an input image to a semantic image with machine-learned regions fortext mapping according to some embodiments.

FIG. 9 depicts a diagrammatic representation of a distributed networkcomputing environment where embodiments disclosed can be implemented.

DETAILED DESCRIPTION

The invention and the various features and advantageous details thereofare explained more fully with reference to the non-limiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well-known starting materials,processing techniques, components, and equipment are omitted so as notto unnecessarily obscure the invention in detail. It should beunderstood, however, that the detailed description and the specificexamples, while indicating some embodiments of the invention, are givenby way of illustration only and not by way of limitation. Varioussubstitutions, modifications, additions, and/or rearrangements withinthe spirit and/or scope of the underlying inventive concept will becomeapparent to those skilled in the art from this disclosure.

Referring to FIG. 1, a diagrammatic representation of a dataclassification and extraction system 100 according to some embodimentsis shown. As depicted in FIG. 1, system 100 can include an OCR module110, a semantics converter module 120, a language knowledge base module130, and an R-CNN run time engine 140.

An input image 101 can be provided to or received by OCR module 110. OCRmodule 110 is configured for identifying characters that are depictedwithin input image 101 and producing an output 115 that includes theidentified characters and corresponding geometries. As used herein,“geometries” refers to the position information associated with thecharacters. The position information identifies where particularcharacters are found in the image. As illustrated in FIG. 1, theposition information can be represented in output 115 as bounding boxes,frames, rectangles, or the like. Output 115 can be a text document thatcontains the text of the input image and corresponding positioninformation. When input image 101 already contains such a text document,the OCR operation performed by OCR module 110 can be skipped.

In some embodiments, the text document can be provided by OCR module 110to semantics converter module 120. In some embodiments, the textdocument can be provided to semantics converter module 120 through meansother than an OCR system.

In some embodiments, semantics converter module 120 is configured foridentifying semantic types associated with semantic information in thetext document. In some embodiments, semantics converter module 120 doesso by accessing language knowledge base module 130 using any suitablelook-up method (e.g., through an inverted index, reverse index,dictionary, etc.).

Referring to FIG. 2, a diagrammatic representation of the informationthat may be contained in a language knowledge base 230 according to someembodiments is shown. Language knowledge base 230 can includeinformation identifying various different semantic types that may beassociated with text found in an image. These semantic types mayinclude, for example, common nouns, common verbs, city names, zip codes,streets, month names, person names, numbers, separators, currencysymbols, articles, prepositions, and the like. These types of text maybe found in almost any document image. “*” denotes non-text types thatmay be supported by an OCR engine.

Initially, lists of possible values (e.g., character strings) for eachsemantic type can be entered into language knowledge base 230. Overtime, language knowledge base 230 can be augmented with additionalentries, for instance, automatically as the system is trained and used.

In some embodiments, language knowledge base 230 can include semantictypes that are specific to a particular use or user, such asdomain-specific nouns and domain-specific verbs. This allows a semanticsconverter (e.g., semantics converter module 120) to identify semantictypes associated with text that is not usually found in typicalenterprise documents. For instance, an enterprise may use certainterminology or acronyms that are not typically used by otherenterprises, and whose semantic type(s) might be unclear outside thecontext of the enterprise or the industry of which the enterprise is in.The entries in a language knowledge base for domain-specific semantictypes may be supplemented by manually adding these entries to thelanguage knowledge base, or through training and using the system. Insome embodiments, language knowledge base 230 can be configured toinclude non-textual semantic types such as graphics (e.g., lines,pictures, logos, etc.), layouts, signatures, handprints, etc.

Referring to FIG. 1, language knowledge base 130 can implement anembodiment of language knowledge base 230. In operation of system 100,semantics converter module 120 can look up individual pieces of the text(i.e., character strings in the text document) in language knowledgebase module 130 and determine from language knowledge base module 130which semantic type is associated with each piece of the text. Forinstance, “Smith”, “Jones”, “Mary”, and the like may be associated withthe semantic type “person name”. Similarly, “Texas”, “Vermont”,“Colorado”, etc. may be associated with the semantic type “state name”;“-”, “/” and other characters may be associated with the semantic type“separator”; and so on. In some embodiments, semantic types can beassociated with non-textual information such as layout informationand/or graphics (e.g., lines, pictures, logos, etc.) in additional totextural information.

In some embodiments, the semantic types in the text document can berepresented in a non-textual manner using a coding scheme. For example,suppose text strings such as “John Smith”, “Austin”, “Texas”, “512”,“music”, “the”, “-”, and “!” are determined by semantics convertermodule 120 as having semantic types “name”, “city”, “state”, “areacode”, “keyword”, “article”, “separator”, and “punctuation”. Rather thanextracting the actual content of the text strings, semantics convertermodule 120 is operable to determine a code for each text string by itssemantic type. For example, “name” may be coded as “1” or “yellow” witha set of x-y coordinates (relative to the document); “city” may be codedas “2” or “blue” with another set of x-y coordinates; “state” may becoded as “3” or “purple” with yet another set of x-y coordinates; and soon. Other coding schemes suitable for identifying semantic informationby type may also be used.

Semantics converter module 120 can generate a semantic image 125 withthe coded semantic information. As illustrated in FIG. 1, semanticsconverter module 120 can utilize the position information (e.g.,geometric coordinates) of the text strings in the text document toposition, in semantic image 125, the coded semantic informationdetermined from the text strings. However, semantics converter module120 does not include the actual content of the text document or thesemantic types derived from the text document in semantic image 125. Forexample, when a color coding scheme is utilized to code semanticinformation, semantic image 125 can include blocks of colors, each colorrepresenting a coded semantic type derived from the text document. Inthis example, semantic image 125 does not have any textual information.A semantic image can be, but is not required to be, a pixel image. Inthis disclosure, a semantic image refers to an image that contains codedsemantic information (e.g., represented in blocks) corresponding toassociated semantic types and geometrics of text in an original documentfrom which the semantic image is generated.

As illustrated in FIG. 1, semantic image 125 generated by semanticsconverter module 120 is provided as input to an R-CNN runtime engine140. In some embodiments, R-CNN runtime engine 140 can be a ML enginethat implements an R-CNN with a DL feedback loop that produces R-CNNparameters 145. Before describing R-CNN runtime engine 140 in detail, anoverview of ML, DL, and CNN might be helpful.

ML and DL are artificial intelligence (AI) learning methods, with DLbeing a subfield of ML. A ML engine uses algorithms to parse data, learnfrom the parsed data, and make informed decisions based on what it haslearned. An algorithm can have algorithmic rules but is not a completecomputer program with a set of instructions that can be translated orexecuted by a processor. Rather, an algorithm is a limited sequence ofsteps for solving a problem such as document processing. For instance, aML engine can be trained to parse text from a document and determine aspecific document class and/or extract specific fields like addresses,dates, amounts, etc. based on what it has learned from the text. Thisprocess, however, requires that the ML engine be trained to recognizeeach piece of text in the document and all the possible variants of thepiece of text.

For example, to recognize a piece of text “July 31, 2018”, the ML enginewould need to be trained to recognize “July” out of all twelve months,“31” out of all possible days in a month, and “2018” out of a range ofyears. Even when the range of years is limited to 200 and consideringonly two separators: “ ” and “,”, the ML engine would need to be trainedto recognize 148,000 combinations. The number of training samples canincrease significantly when considering additional separators (e.g.,“.”, “-”, “/”, “|”, etc.) and/or formats (e.g., “7”, “july”, “07”, “r”,“7th”, etc.). Because the ML engine can break down a problem into parts,solve each part in order, and then combine the solved parts to create ananswer to the problem, it does not require powerful processors and canrun on low-end computing devices.

DL utilizes a different approach. Unlike standard ML algorithms thatbreak problems down into parts and solve them individually, DL solveseach problem from beginning to end and thus requires high-end machineswith graphics cards capable of crunching numbers (e.g., graphicprocessors or processing units) at a high speed. Much like how humanbrains work, the more data is fed to a DL algorithm, the better it getsat solving a task over time. Small amounts of data would yield lowerperformance. Thus, DL also requires enormous amounts of sample data.

DL has been studied for use in image processing. As an example, an imagemay be fed to a ML engine implementing a DL algorithm. The ML engine canscan all the pixels within the image to discover edges and shapes thatcan be used to distinguish between objects in the image. The ML enginecan then sort the edges and shapes into a ranked order of possibleimportance to determine what objects are in the image. The ML engine isnot provided with any pre-fed information. Rather, it uses the DLalgorithm to determine how many lines the shapes have, whether the linesare connected, and how the lines are connected (e.g., perpendicular orat an angle). Over time, the DL algorithm learns that a circle does notfit in with a square or a triangle. This “thinking” and learning processrequires powerful machines to process data generated by the DLalgorithm. Accordingly, these machines often reside in data centers tocreate an artificial neural network that can mimic a human neuralnetwork.

A ML engine implementing a DL algorithm can take longer than a typicalML to train because it is learning by example on its own and does notrely on hand-fed data. The amount of time needed to train such a MLengine can depend on the problem that the DL algorithm is tasked tosolve. For example, training a ML engine implementing a DL algorithm fortranslating words spoken in one human language into words written in adifferent language can take years due to the differences in the overalllanguage, language use, voice pitches, and hardware-based capabilities.

Accordingly, DL usually requires a significant number of trainingsamples and long training time. To this end, embodiments disclosedherein provide a solution to combine the powers of ML and DL bysignificantly reducing the number of training samples that would beneeded to train a ML engine implementing a DL algorithm. In doing so,the time needed to train the ML engine can also be significantlyreduced.

In some embodiments, the significant reduction in the number of trainingsamples and time is made possible by transforming text-based documentsinto images while preserving the main semantic information, leveraging alanguage knowledge base (e.g., language knowledge base 230 describedabove). The resulting semantic images can then be input to a ML engineimplementing a DL algorithm. This transformation allows for applicationof image-based DL approaches to document processing such asclassification, data extraction, etc.

As explained above, semantic images need not have any textualinformation. Thus, the ML engine does not need to be trained torecognize textual information (e.g., characters). Rather, the ML enginemay only need to be trained on a few hundred to a few thousand images torecognize an “object”, “shape”, or “geometry” from the semantic images.Here, each such “object”, “shape”, or “geometry” can represent a pieceof coded semantic information.

As described above, in generating a semantic image, text stringsassociated with an input image are converted into semantic information,which is then coded by semantic types. The coded semantic information ispositioned in the semantic image in accordance with geometriesassociated with the text strings. Following the example above, “July”,“31”, and “2018” in an input image would be identified and convertedsemantic information as “month”, “day”, and “year”, which is then codedby a semantic type “date” and positioned in a semantic image inaccordance with geometries associated with “July”, “31”, and “2018”. Thesemantic image thus generated contains an “object”, “shape”, or“geometry” representing the coded semantic information (also referred toherein as a semantic representation). Instead of having to recognize148,000 combinations of “July”, “31”, and “2018”, the ML engine now onlyneeds to be trained to recognize one semantic representation of “July”,“31”, and “2018”. This can be a significant reduction in the number oftraining samples and time required to train the ML engine. Further,because ML engines implementing DL algorithms would run on graphicsprocessing units (GPUs), the processing powers of such ML engines can beincreased due to the huge computational power of GPUs.

Based on the semantic representation(s) that the ML engine has learned,the ML engine can determine a document type (classification) and/or findregions of specific objects such as addresses, dates, amounts, etc.(data extraction). In some embodiments, a specifically trained ANN suchas an R-CNN can be used.

R-CNN is typically used for image segmentation. More specifically, anR-CNN can be trained to take in an image and correctly identify wherethe main objects are in the image. This is accomplished by creatingbounding boxes (or region proposals), selectively searching the imagethrough windows of different sizes and, for each size, trying to grouptogether adjacent pixels by texture, color, or intensity to identifyobjects. Once the region proposals are created, the R-CNN forms eachproposed region to a standard size box, passes the box through apre-trained CNN such as the AlexNet, and runs a support vector machine(SVM) to determine what object is in the box (a region proposal). TheSVM is configured for classifying whether the region proposal is anobject and, if so, what object. Once the region proposal is classified,the R-CNN runs a linear regression model on the region proposal togenerate tighter coordinates for the box. The R-CNN then outputs thebounding boxes and labels for each object detected in the image.

A person of ordinary skill in the art understands how to design anR-CNN. Accordingly, R-CNN is not further described herein. Oncedesigned, the parameters for the R-CNN (“R-CNN parameters”) aredetermined by DL. R-CNN parameters are values and coefficients for theR-CNN. Examples of trained neural network parameters can include: numberof neurons in the hidden layer, learning rate, momentum, training type,epoch, minimum error, etc.

In the example of FIG. 1, R-CNN runtime engine 140 can leverage thepower of R-CNN to detect regions in semantic image 125 by trying togroup together adjacent semantic representations by texture, color,intensity, or a semantic coding scheme. For instance, semantic types“street”, “city”, “state”, and “zip code” may be adjacent to one anotherand are coded in similar textures, colors, or intensities, or within anumerical range. As such, a DL learner (e.g., an R-CNN) may create aregion proposal that groups these semantic representations together. AnSVM can be configured to identify such a region proposal as an“address”. Likewise, a region that has a 1 or 2-digit number, a month,and a four-digit number may be identified as a date.

In some embodiments, regions (e.g., an “address” region, a “date”region, or other regions) in semantic image 125 thus detected by R-CNNruntime engine 140 can be provided as input 155 to text mapping module150, either alone or in combination with other information (e.g., thetext from the document provided to the semantics converter module, orthe semantic information generated by the semantics converter module).In some embodiments, text mapping module 150 can be configured formapping regions identified in input 155 to data structures on a textlayer (e.g., text document 115). For example, suppose input 155 isidentified as an invoice having regions for an address, an invoicenumber, an invoice date, a purchase order number, a table header, atable body, and an invoice total. Text mapping module 150 can map theseregions to the address, the invoice number, the invoice date, thepurchase order number, the table header, the table body, and the invoicetotal in text document 115. A database can be updated to include textdata 160 extracted from the regions. The detection rate and therecognition rate can be at least as good as a rule-based system, butwith significantly reduced manual coding and maintenance.

In some embodiments, R-CNN runtime engine 140 may also extractinformation from semantic image 125 based upon the identified regions(e.g., extracting an address and identifying it as a customer's address,identifying a number as an invoice number, identifying a document type,etc.).

In some embodiments, extracted text 160 can contain well-structured datawith data fields (e.g., “address”, “invoice number”, “invoice date”,“purchase order number”, “table header”, “table body”, and “invoicetotal”) and actual content from text document 115. This allows extractedtext 160 to be used for further processing by a downstream computingfacility or facilities such as a content management system, informationmanagement system, database management system, invoice processingsystem, enterprise resource planning system, etc.

Knowledge of such fields can come from a field type knowledge base suchas one shown in FIG. 3. Unlike a language knowledge base, which can bebased on any document type and require no domain-specific orsubject-matter knowledge, a field type knowledge base can be configuredto contain higher-level, domain-specific knowledge. In the non-limitingexample shown in FIG. 3, field type knowledge base 370 is configured foraggregating field-specific reference data in relation to semantic imagesgenerated from images of invoices. Accordingly, field type knowledgebase 370 can contain information on fields specific to the domain ofinvoice processing. As illustrated in FIG. 4, field-specific referencedata contained in a field type knowledge base (e.g., field typeknowledge base 470) can be provided in a training process 400 as inputfor training R-CNN 480. Trained R-CNN 480 can then output R-CNNparameters 445 which, as described above, can be input to a ML engine(e.g., R-CNN runtime engine 140). The ML engine, in turn, can improveits performance and operation using R-CNN parameters 445.

In some embodiments, the semantic image generated by the semanticsconverter module described above may also be used by the ML engine(e.g., R-CNN runtime engine 140) to facilitate training of the R-CNN.Typically, training of a ML engine does not utilize a DL feedback loopor semantic image. This is illustrated in a ML training process 500shown in FIG. 5.

In the example of FIG. 5, a processor 502 (e.g., a ML engine) can outputa learned result (e.g., a document classification) and a confidencelevel for the learned result. If the confidence level meets a minimumthreshold (504), the output is provided to a downstream computingfacility for further processing (506). Otherwise, the learned result isfed to a training process (508) with explicit or implicit learning(explained below). For example, suppose the minimum threshold is 99% andprocessor 502 is 98.7% confident that the document classification iscorrect for an input document. In this case, because the confidencelevel does not meet the minimum threshold, the learned result is sent totraining process 508 so that a knowledge base 510 used by processor 502can be updated.

As illustrated in FIG. 5, results from a ML engine are typicallyprovided for further processing or used to update a knowledge base. Insome embodiments, a ML engine can be enhanced with a DL feedback loopthat includes training using semantic images. This is illustrated inFIG. 6.

In the example of FIG. 6, a semantic image 625 is processed by aprocessor 640 (e.g., a ML engine). Processor 640 can implement anembodiment of R-CNN runtime engine 140 described above. Processor 640 isconfigured for processing semantic image 625, locating regions ofinterest, classifying or otherwise identifying regions in semantic image625, and outputting an R-CNN result and a confidence level (which can be0-100%). The confidence level is compared with a threshold (685). If theconfidence level meets or exceeds the threshold, the R-CNN result isprovided to a downstream computing facility for further processing(690). Otherwise, the R-CNN result is sent to a DL feedback loop 600.

In some embodiments, DL feedback loop 600 can include training process682. In some embodiments, training process 682 can be implicitly orexplicitly based on human knowledge. For example, suppose semantic image625 is generated from an input image of an invoice and a user may betasked with validating information contained in the invoice. Forinstance, the user may be tasked with entering and/or validating data inthe invoice (e.g., in a workflow). Such a validation can be done, forexample, through a single click entry where the user places a pointingdevice such as a mouse over an address in a document image of theinvoice and all the textual information is captured and entered into aform (e.g., a database form, a web form, etc.) which is to be filledout. At this point, the user is creating reference data for DL withoutknowing that the validation that the user is doing is actuallyperforming this referencing of data. This validation can be a separateprocess in which all the addresses of all the documents are processed.As described above, for each address, the position information where theaddress is located on the document is also preserved. This locationinformation can be mapped to the semantic image. Accordingly,unbeknownst to the user, field-specific reference data from thevalidation can be used in training process 682. This is referred to asimplicit training. Unlike implicit training, explicit training is notunsupervised or unattended and the user knows that s/he is part oftraining process 682. The source of field-specific reference data needsnot be a human and can be an enterprise repository where taggedreference data is stored.

Unsupervised learning refers to a ML task of learning a function fromthe structure of unlabeled data. Many algorithms, including ANNs, can beused for unsupervised learning. Supervised learning refers to a ML taskof learning a function that maps an input to an output based on exampleinput-output pairs. This ML task requires labeled training data so thatthe function can be inferred from the labeled training data.Unsupervised learning and supervised learning are known to those skilledin the art and thus are not further described herein.

In some embodiments, semantic image 625 and field-specific referencedata (obtained implicitly or explicitly through training process 682)can be used to update a field type knowledge base 670. Field typeknowledge base 670 can implement an embodiment of field type knowledgebase 370 described above.

In some embodiments, a learner 680 implementing a DL algorithm can learnfrom the differences between the actual result (e.g., manually correctedand validated field data for semantic image 625) and the expected result(e.g., the R-CNN result produced by processor 640 from semantic image625). Because a semantic image is provided to learner 680, the DLalgorithm only has to analyze and learn the semantic types in thesemantic image, rather than the actual content.

Consider an example in which an ANN is designed to recognize addresses.The ANN must be able to identify all of the potential street addresses,all of the potential cities, all of the potential states, and all of thepotential zip codes that might appear in the address. Conventionally,such an ANN would be trained to recognize each of these itemsseparately—an astronomical number of possibilities. However, in theexample of FIG. 6, a semantic image, and not a text document, isprovided to the R-CNN (in learner 680). Accordingly, the R-CNN only hasto be trained to recognize each semantic type in the semantic image.Consequently, the R-CNN only needs to learn a fraction of thepossibilities. For instance, instead of learning to recognize all 50states (in the US alone), the R-CNN only has to be trained to recognizethe “state name” semantic type. This greatly reduces the number ofexamples that are necessary to train the R-CNN, and likewisesignificantly reduces the amount of time required for the training.

Once trained, learner 680 is operable to output R-CNN parameters 645.This is the output of DL feedback loop 600. As illustrated in FIG. 1,the output from such a DL feedback loop can be fed back into and used byan R-CNN runtime engine. Although the number of layers in an R-CNN isdefined initially, the first time when the R-CNN runtime engine is run,there are no such parameters which can be applied. After the R-CNNruntime engine has been running for a while (after collecting data andperforming the learning step described above), the R-CNN parameters willbe filled and can be applied to the R-CNN runtime engine. Further, theknowledge of the field type knowledge base in terms of the parameterscan be applied to the R-CNN runtime engine. In this way, the R-CNNruntime engine can become more powerful over time (e.g., in just oneday).

Referring to FIG. 7, a flowchart illustrating an exemplary method inaccordance with some embodiments is shown. In this method, an inputimage is first received (701). As a non-limiting example, the inputimage can be a pixel image. In such a case, the pixel image can consistan array of pixels, each of which has a corresponding color or shade ofgray. If a corresponding text layer or document does not already exist,the input image is processed, for instance, utilizing an OCR technique,to obtain textual information with geometries from the input image(705). In some cases, a text layer or document may already be providedwith the input image. For example, portable document format (PDF) is afile format that can capture all the elements of a printed document asan electronic image. Thus, a PDF fie can contain not only images, butalso text, interactive buttons, hyperlinks, embedded fonts, video, etc.When viewed by an end user, the text document may look nearly identicalto the original input image, although this is not necessary in allembodiments. The text document can include text and geometries, orlocation information that identifies where in the document theidentified text can be found.

Next, semantic information is determined from the textual information(710). In some embodiments, this can involve examining the text documentto identify individual character strings within the text and, for eachof these individual character strings, accessing a language knowledgebase and looking up the semantic type associated with each of theindividual character strings.

A semantic image corresponding to the original input image and thecorresponding text document can then be generated (715). The semanticimage includes the semantic information that was obtained from thelanguage knowledge base. Character strings that have the same semantictype may be considered to be in a single semantic “layer” of the image.For instance, all names of people can be in a “person name” layer, allstreet names can be a “street name” layer, and so on. The semanticinformation may be included in the semantic image in a way that isvisible to a user, such as color-coding the image to represent thesemantic types. Alternatively or additionally, the semantic-typeinformation may be encoded in the document in a way that is not visuallyrepresentable.

The semantic image can then be provided to a ML engine for processing(720). In some embodiments, the ML engine implements an R-CNN that canlearn to extract regions from semantic images as it processes eachsemantic image, thereby continually improving its own performance. Inthis case, the R-CNN examines the received semantic image and identifiesregions within the semantic image. For example, if the R-CNN detectscertain features of the semantic image (e.g., coded semanticinformation) are in close proximity to each other and are codedsimilarly or close to each other (e.g., similar colors, shades,intensities, etc.), the R-CNN may group these features as a region for afield, using knowledge (e.g., the field-specific reference datadescribed above) from a field knowledge base. In some cases, if anumber-type semantic information is close to a keyword-type semanticinformation (e.g., “invoice”), the region may be determined to representa field (or a variable) and its corresponding value (e.g., the keyword“invoice” and the corresponding invoice number).

The regions in the semantic image thus identified can be mapped to thetext document and data from the regions can be extracted in from thetext document a structured manner (725). The data extracted from theregions can be used to update a database and/or provided to a downstreamcomputing facility for further processing. In the example of an addressregion, the separate character strings representing the number, streetname, city, state, and zip code can be collectively interpreted as asingle address, rather than as separate and independent pieces of text.This address may further be interpreted as being associated with aparticular person who is identified by nearby character strings having“person name” semantic types.

The transformation of an input image to a text document, to a semanticimage, then to a document with identified regions is illustrated inFIGS. 8A-8E. FIG. 8A depicts an input image 801 that is provided to asystem implementing a method disclosed herein. FIG. 8B depicts a textdocument 815 that contains text with geometries. FIG. 8C depicts asemantic image 825 with coded semantic information and without textualinformation (or any graphics or layout information). FIG. 8D depicts animage 835 that overlays the coded semantic information and regions ofinterest determined by an R-CNN.

In the example of FIG. 8A, input image 801 shows an invoice. If inputimage 801 is a pixel image, input image 801 would contain imageinformation only—each pixel has a corresponding color or grayscalevalue, but no textual information other than an image of the text. Sucha pixel image could, for example, be captured by scanning orphotographing a paper copy of the document.

After the pixel image is processed by an OCR module (e.g., OCR module110), a text document is produced. FIG. 8B shows an example of textdocument 815. The text document may be in a variety of formats, such asa PDF file, a Word document, etc. The text document in this examplelooks nearly identical to the pixel image of FIG. 4A, but the textdocument contains textual information that is encoded as text, ratherthan pixels which depict an image of text. In addition to the encodedtextual information, the text document includes geometries associatedwith the text. For example, the character string “INVOICE” is associatedwith a specific position at the upper left portion of the document,while the character strings “total” and “GBP” are associated withcorresponding positions at the lower right portion of the document. Theposition information (geometries) of the text may or may not be visibleto an end user. In the example of FIG. 8B, the position information(e.g., geometrics represented by boxes) is visible for each piece oftext (e.g., a character string) in text document 815.

Text document 815 can be processed (e.g., by semantics converter module120) to identify the semantic information and corresponding semantictypes associated with the text in the document. After the semantic typesare identified, semantic image 825, which is depicted in FIG. 8C, can begenerated as described above. The semantic image shows coded semantictypes for the text in the document. As a non-limiting example, semanticimage 825 shows a block at the position of each character string. Theblock can have a color, shade, intensity, texture, or any suitablesemantic code corresponding to the semantic type of the characterstring. For example, the “INVOICE” character string may be determined tobe a keyword, so a block that has a shade corresponding to the “keyword”semantic type is displayed in semantic image 825 at the positionoccupied by the word “INVOICE” in text document 815. The shadecorresponding to the “keyword” semantic type may be a particular color,a particular shade of gray, etc. Semantic image 825 can include a blockof this same shade at the position of each keyword in the document.Character strings that are associated with each unique semantic type canbe represented in semantic image 825 by a block of a particular shadecorresponding to this unique semantic type. For example, numericcharacter strings may be represented by yellow, separators may berepresented by red, city names may be represented by blue, and so on.

Again, as described above, semantic image 825 depicted in FIG. 8C caninclude only the shaded blocks that represent the semantic types of thecorresponding character strings. Semantic image 825 does not include thecharacter strings themselves. In alternative embodiments, it is possibleto include both the semantic type information and the character stringsin a semantic image. The coded semantic information that is provided bysemantic image 825 in the example of FIG. 8C is sufficient to allow anR-CNN to identify regions of interest within semantic image 825. Forexample, the R-CNN can identify the geometric region of characterstrings that includes a number, street name, city, state, and zip codeas an address without having to know the specific street number, streetname, city, state, or zip code.

Image 835 shown in FIG. 8D illustrates regions of interest that can beidentified by the R-CNN from semantic image 825 shown in FIG. 8C. Inthis example, these regions include a purchase order number region 802,an invoice number region 804, an invoice date region 806, an addressregion 808, a table header region 810, a table body region 812, and aninvoice totals region 814. While FIG. 8D shows that the identifiedregions overlay the coded semantic information, this is a visualizationand not a programmatic output of the R-CNN. FIG. 8E depicts an exampleimage 855 with just the regions thus extracted.

The solution described above can be usable for document processing,e.g., classification and data extraction. Accordingly, an additionalmapping step (e.g., by text mapping module 150 of FIG. 1) can beperformed to map the regions to the text in the text document andextract data from their respective regions. Further, the mapping stepmay include classifying the input image based on the identified regions.For example, purchase order number region 802, invoice number region804, invoice date region 806, address region 808, table header region810, table body region 812, and invoice totals region 814 together mayindicate that input image 801 is an invoice. Accordingly, the textmapping module may classify input image 801 as an invoice, associate theinvoice with the invoice number, the invoice date, and the purchaseorder number extracted from the invoice, and update a database toinclude an entry of the invoice, including a customer's name andaddress, as well as the specific line item descriptions and costsspecified in the table body, and the total amount of the invoice.

FIG. 9 depicts a diagrammatic representation of a distributed networkcomputing environment where embodiments disclosed herein can beimplemented. In the example illustrated, a network computing environment900 can include a network 914 bi-directionally coupled to a computer912, a computer 915, and a computer 916. Computer 916 can bebi-directionally coupled to a data store or database 918. Network 914may represent a combination of wired and wireless networks that networkcomputing environment 900 may utilize for various types of networkcommunications known to those skilled in the art.

For purpose of illustration, a single system is shown for each ofcomputer 912, computer 915, and computer 916. However, with each ofcomputer 912, computer 915, and computer 916, a plurality of computers(not shown) may be interconnected to each other over network 914. Forexample, a plurality of computers 912 and a plurality of computers 915may be coupled to network 914. Computers 912 may include data processingsystems for communicating with computer 916. Computers 915 may includedata processing systems for individuals whose jobs may require them toconfigure services used by computers 912 in network computingenvironment 900.

First enterprise computer 912 can include central processing unit(“CPU”) 920, read-only memory (“ROM”) 922, random access memory (“RAM”)924, hard drive (“HD”) or storage memory 926, and input/output device(s)(“I/O”) 928. I/O 929 can include a keyboard, monitor, printer,electronic pointing device (e.g., mouse, trackball, stylus, etc.), orthe like. Computer 912 can include a desktop computer, a laptopcomputer, a personal digital assistant, a cellular phone, or nearly anydevice capable of communicating over a network. Computer 915 may besimilar to computer 912 and can comprise CPU 950, ROM 952, RAM 954, HD956, and I/O 958.

Likewise, computer 916 may include CPU 960, ROM 962, RAM 964, HD 966,and I/O 968. Computer 916 may include one or more backend systemsconfigured for providing a variety of services to computers 912 overnetwork 914. These services may utilize data stored in data store 918.Many other alternative configurations are possible and known to skilledartisans.

Each of the computers in FIG. 9 may have more than one CPU, ROM, RAM,HD, I/O, or other hardware components. For the sake of brevity, eachcomputer is illustrated as having one of each of the hardwarecomponents, even if more than one is used. Each of computers 912, 915,and 916 is an example of a data processing system. ROM 922, 952, and962; RAM 924, 954, and 964; HD 926, 956, and 966; and data store 918 caninclude media that can be read by CPU 920, 950, or 960. Therefore, thesetypes of memories include non-transitory computer-readable storagemedia. These memories may be internal or external to computers 912, 915,or 916.

Portions of the methods described herein may be implemented in suitablesoftware code that may reside within ROM 922, 952, or 962; RAM 924, 954,or 964; or HD 926, 956, or 966. In addition to those types of memories,the instructions in an embodiment disclosed herein may be contained on adata storage device with a different computer-readable storage medium,such as a hard disk. Alternatively, the instructions may be stored assoftware code elements on a data storage array, magnetic tape, floppydiskette, optical storage device, or other appropriate data processingsystem readable medium or storage device.

Those skilled in the relevant art will appreciate that the invention canbe implemented or practiced with other computer system configurations,including without limitation multi-processor systems, network devices,mini-computers, mainframe computers, data processors, and the like. Theinvention can be embodied in a computer or data processor that isspecifically programmed, configured, or constructed to perform thefunctions described in detail herein. The invention can also be employedin distributed computing environments, where tasks or modules areperformed by remote processing devices, which are linked through acommunications network such as a local area network (LAN), wide areanetwork (WAN), and/or the Internet. In a distributed computingenvironment, program modules or subroutines may be located in both localand remote memory storage devices. These program modules or subroutinesmay, for example, be stored or distributed on computer-readable media,including magnetic and optically readable and removable computer discs,stored as firmware in chips, as well as distributed electronically overthe Internet or over other networks (including wireless networks).Example chips may include Electrically Erasable Programmable Read-OnlyMemory (EEPROM) chips. Embodiments discussed herein can be implementedin suitable instructions that may reside on a non-transitorycomputer-readable medium, hardware circuitry or the like, or anycombination and that may be translatable by one or more server machines.Examples of a non-transitory computer-readable medium are provided belowin this disclosure.

ROM, RAM, and HD are computer memories for storing computer-executableinstructions executable by the CPU or capable of being compiled orinterpreted to be executable by the CPU. Suitable computer-executableinstructions may reside on a computer-readable medium (e.g., ROM, RAM,and/or HD), hardware circuitry or the like, or any combination thereof.Within this disclosure, the term “computer-readable medium” is notlimited to ROM, RAM, and HD and can include any type of data storagemedium that can be read by a processor. Examples of computer-readablestorage media can include, but are not limited to, volatile andnon-volatile computer memories and storage devices such as random accessmemories, read-only memories, hard drives, data cartridges, directaccess storage device arrays, magnetic tapes, floppy diskettes, flashmemory drives, optical data storage devices, compact-disc read-onlymemories, and other appropriate computer memories and data storagedevices. Thus, a computer-readable medium may refer to a data cartridge,a data backup magnetic tape, a floppy diskette, a flash memory drive, anoptical data storage drive, a CD-ROM, ROM, RAM, HD, or the like.

The processes described herein may be implemented in suitablecomputer-executable instructions that may reside on a computer-readablemedium (for example, a disk, CD-ROM, a memory, etc.). Alternatively, thecomputer-executable instructions may be stored as software codecomponents on a direct access storage device array, magnetic tape,floppy diskette, optical storage device, or other appropriatecomputer-readable medium or storage device.

Any suitable programming language can be used to implement the routines,methods or programs of embodiments of the invention described herein,including C, C++, Java, JavaScript, HTML, or any other programming orscripting code, etc. Other software/hardware/network architectures maybe used. For example, the functions of the disclosed embodiments may beimplemented on one computer or shared/distributed among two or morecomputers in or across a network. Communications between computersimplementing embodiments can be accomplished using any electronic,optical, radio frequency signals, or other suitable methods and tools ofcommunication in compliance with known network protocols.

Different programming techniques can be employed such as procedural orobject oriented. Any particular routine can execute on a single computerprocessing device or multiple computer processing devices, a singlecomputer processor or multiple computer processors. Data may be storedin a single storage medium or distributed through multiple storagemediums, and may reside in a single database or multiple databases (orother data storage techniques). Although the steps, operations, orcomputations may be presented in a specific order, this order may bechanged in different embodiments. In some embodiments, to the extentmultiple steps are shown as sequential in this specification, somecombination of such steps in alternative embodiments may be performed atthe same time. The sequence of operations described herein can beinterrupted, suspended, or otherwise controlled by another process, suchas an operating system, kernel, etc. The routines can operate in anoperating system environment or as stand-alone routines. Functions,routines, methods, steps and operations described herein can beperformed in hardware, software, firmware or any combination thereof.

Embodiments described herein can be implemented in the form of controllogic in software or hardware or a combination of both. The controllogic may be stored in an information storage medium, such as acomputer-readable medium, as a plurality of instructions adapted todirect an information processing device to perform a set of stepsdisclosed in the various embodiments. Based on the disclosure andteachings provided herein, a person of ordinary skill in the art willappreciate other ways and/or methods to implement the invention.

It is also within the spirit and scope of the invention to implement insoftware programming or code any of the steps, operations, methods,routines or portions thereof described herein, where such softwareprogramming or code can be stored in a computer-readable medium and canbe operated on by a processor to permit a computer to perform any of thesteps, operations, methods, routines or portions thereof describedherein. The invention may be implemented by using software programmingor code in one or more digital computers, by using application specificintegrated circuits, programmable logic devices, field programmable gatearrays, optical, chemical, biological, quantum or nanoengineeredsystems, components and mechanisms may be used. The functions of theinvention can be achieved by distributed or networked systems.Communication or transfer (or otherwise moving from one place toanother) of data may be wired, wireless, or by any other means.

A “computer-readable medium” may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, system ordevice. The computer-readable medium can be, by way of example only butnot by limitation, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, system, device,propagation medium, or computer memory. Such computer-readable mediumshall generally be machine readable and include software programming orcode that can be human readable (e.g., source code) or machine readable(e.g., object code). Examples of non-transitory computer-readable mediacan include random access memories, read-only memories, hard drives,data cartridges, magnetic tapes, floppy diskettes, flash memory drives,optical data storage devices, compact-disc read-only memories, and otherappropriate computer memories and data storage devices. In anillustrative embodiment, some or all of the software components mayreside on a single server computer or on any combination of separateserver computers. As one skilled in the art can appreciate, a computerprogram product implementing an embodiment disclosed herein may compriseone or more non-transitory computer-readable media storing computerinstructions translatable by one or more processors in a computingenvironment.

A “processor” includes any, hardware system, mechanism or component thatprocesses data, signals or other information. A processor can include asystem with a central processing unit, multiple processing units,dedicated circuitry for achieving functionality, or other systems.Processing need not be limited to a geographic location, or havetemporal limitations. For example, a processor can perform its functionsin “real-time”, “offline”, in a “batch mode”, etc. Portions ofprocessing can be performed at different times and at differentlocations, by different (or the same) processing systems.

As used herein, the terms “comprises”, “comprising”, “includes”,“including”, “has”, “having”, or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,product, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, product, article, orapparatus.

Furthermore, the term “or” as used herein is generally intended to mean“and/or” unless otherwise indicated. For example, a condition A or B issatisfied by any one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present). As used herein, a termpreceded by “a” or “an” (and “the” when antecedent basis is “a” or “an”)includes both singular and plural of such term, unless clearly indicatedotherwise (i.e., that the reference “a” or “an” clearly indicates onlythe singular or only the plural). Also, as used in the descriptionherein, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application.Additionally, any signal arrows in the drawings/figures should beconsidered only as exemplary, and not limiting, unless otherwisespecifically noted. The scope of the present disclosure should bedetermined by the following claims and their legal equivalents.

What is claimed is:
 1. A method, comprising: determining, by a computerutilizing a language knowledge base, semantic types associated withtextual content found in an image; generating, by the computer, asemantic image of the textual content, the semantic image havingsemantic representations of the textual content found in the image,wherein a semantic representation includes positioning information for apiece of text represented by the semantic representation and a codecorresponding to a semantic type of the piece of text; providing, by thecomputer, the semantic image as input to a machine learning engine thatimplements an artificial neural network (ANN), the ANN trained torecognize the semantic representations and extract regions from thesemantic image utilizing the semantic representations; and providing, bythe computer, the regions extracted from the semantic image as input toa text mapping module, the text mapping module operable to extract datafrom the image utilizing the regions extracted from the semantic image.2. The method according to claim 1, wherein the ANN is operable to learnfrom the semantic representations and output a learned result withoutrelying on the textual content found in the image, the learned resultcomprising a classification for the image.
 3. The method according toclaim 2, further comprising: determining whether a confidence level forthe learned result meets a threshold; and responsive to the confidencelevel for the learned result not meeting the threshold, providing thelearned result to a training process so as to update the languageknowledge base.
 4. The method according to claim 1, wherein the ANNcomprises a region-based convolutional neural network (R-CNN) andwherein the machine learning engine operates as an R-CNN runtime engine.5. The method according to claim 4, further comprising: providingoutputs from the R-CNN runtime engine to a deep learning (DL) module,the DL module implementing a DL algorithm for analyzing the semantictypes in the semantic image, learning from differences between theoutputs from the R-CNN runtime engine and actual results containingcorrected data from the image, and producing R-CNN parameters thatreflect the differences thus learned.
 6. The method according to claim5, further comprising: providing the R-CNN parameters to the R-CNNruntime engine so as to improve performance of the R-CNN runtime engine.7. The method according to claim 1, wherein the semantic image comprisesa plurality of semantic layers, each of the plurality of semantic layerscorresponding to a semantic type of the semantic types associated withtextual content found in the image.
 8. A system, comprising: aprocessor; a non-transitory computer-readable medium; and storedinstructions translatable by the processor for: determining, utilizing alanguage knowledge base, semantic types associated with textual contentfound in an image; generating a semantic image of the textual content,the semantic image having semantic representations of the textualcontent found in the image, wherein a semantic representation includespositioning information for a piece of text represented by the semanticrepresentation and a code corresponding to a semantic type of the pieceof text; providing the semantic image as input to a machine learningengine that implements an artificial neural network (ANN), the ANNtrained to recognize the semantic representations and extract regionsfrom the semantic image utilizing the semantic representations; andproviding the regions extracted from the semantic image as input to atext mapping module, the text mapping module operable to extract datafrom the image utilizing the regions extracted from the semantic image.9. The system of claim 8, wherein the ANN is operable to learn from thesemantic representations and output a learned result without relying onthe textual content found in the image, the learned result comprising aclassification for the image.
 10. The system of claim 9, wherein thestored instructions are further translatable by the processor for:determining whether a confidence level for the learned result meets athreshold; and responsive to the confidence level for the learned resultnot meeting the threshold, providing the learned result to a trainingprocess so as to update the language knowledge base.
 11. The system ofclaim 8, wherein the ANN comprises a region-based convolutional neuralnetwork (R-CNN) and wherein the machine learning engine operates as anR-CNN runtime engine.
 12. The system of claim 11, wherein the storedinstructions are further translatable by the processor for: providingoutputs from the R-CNN runtime engine to a deep learning (DL) module,the DL module implementing a DL algorithm for analyzing the semantictypes in the semantic image, learning from differences between theoutputs from the R-CNN runtime engine and actual results containingcorrected data from the image, and producing R-CNN parameters thatreflect the differences thus learned.
 13. The system of claim 12,wherein the stored instructions are further translatable by theprocessor for: providing the R-CNN parameters to the R-CNN runtimeengine so as to improve performance of the R-CNN runtime engine.
 14. Thesystem of claim 8, wherein the semantic image comprises a plurality ofsemantic layers, each of the plurality of semantic layers correspondingto a semantic type of the semantic types associated with textual contentfound in the image.
 15. A computer program product comprising anon-transitory computer-readable medium storing instructionstranslatable by a processor for: determining, utilizing a languageknowledge base, semantic types associated with textual content found inan image; generating a semantic image of the textual content, thesemantic image having semantic representations of the textual contentfound in the image, wherein a semantic representation includespositioning information for a piece of text represented by the semanticrepresentation and a code corresponding to a semantic type of the pieceof text; providing the semantic image as input to a machine learningengine that implements an artificial neural network (ANN), the ANNtrained to recognize the semantic representations and extract regionsfrom the semantic image utilizing the semantic representations; andproviding the regions extracted from the semantic image as input to atext mapping module, the text mapping module operable to extract datafrom the image utilizing the regions extracted from the semantic image.16. The computer program product of claim 15, wherein the ANN isoperable to learn from the semantic representations and output a learnedresult without relying on the textual content found in the image, thelearned result comprising a classification for the image.
 17. Thecomputer program product of claim 16, wherein the instructions arefurther translatable by the processor for: determining whether aconfidence level for the learned result meets a threshold; andresponsive to the confidence level for the learned result not meetingthe threshold, providing the learned result to a training process so asto update the language knowledge base.
 18. The computer program productof claim 15, wherein the ANN comprises a region-based convolutionalneural network (R-CNN) and wherein the machine learning engine operatesas an R-CNN runtime engine.
 19. The computer program product of claim18, wherein the instructions are further translatable by the processorfor: providing outputs from the R-CNN runtime engine to a deep learning(DL) module, the DL module implementing a DL algorithm for analyzing thesemantic types in the semantic image, learning from differences betweenthe outputs from the R-CNN runtime engine and actual results containingcorrected data from the image, and producing R-CNN parameters thatreflect the differences thus learned.
 20. The computer program productof claim 19, wherein the instructions are further translatable by theprocessor for: providing the R-CNN parameters to the R-CNN runtimeengine so as to improve performance of the R-CNN runtime engine.